The approaches described in this section could be pursued, but are not necessarily approaches that have previously been conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
An increasing number of network appliances, physical and virtual, are deployed in communication networks such as wide area networks (WAN). For each network appliance, it may be desirable to monitor attributes and statistics of the data traffic handled by the device. For example, information can be collected regarding source IP addresses, destination IP addresses, traffic type, port numbers, etc. for the traffic that passes through the network appliance. Typically this information is collected for each data flow using industry standards such as NetFlow and IPFIX. The collected data is transported across the network to a collection engine, stored in a database, and can be utilized for running queries and generating reports regarding the network.
Since there can be any number of data flows processed by a network appliance each minute (hundreds, thousands, or even millions), this results in a large volume of data that is collected each minute, for each network appliance. As the number of network appliances in a communication network increases, the amount of data generated can quickly become unmanageable. Moreover, transporting all of this data across the network from each network appliance to the collection engine can be a significant burden, as well as storing and maintaining a database with all of the data. Further, it may take longer to run a query and generate a report since the amount of data to be processed and analyzed is so large.
Thus, there is a need for a more efficient mechanism for collecting and storing network traffic statistics for a large number of network appliances in a communication network.